Remote hosting of online, fast-action, interactive video games and other high-end video applications typically requires very low latencies. For example, for twitch video games and applications, low round-trip latency, as measured from the time a user's control input is sent to the hosting service center to the time that the newly generated video content appears on the screen of the user's client device, is typically required. At higher latencies, performance suffers noticeably. Achieving such low latencies over the Internet or other similar networks requires the video compressor at the hosting service to generate a packet stream with particular characteristics such that the packet sequence flowing through the entire path from the hosting service to the client device is not subject to delays or excessive packet loss. In addition, the video compressor must create a packet stream which is sufficiently robust so that it can tolerate the inevitable packet loss and packet reordering that occurs in normal Internet and network transmissions.
In streaming video technologies, lost or dropped packets can result in highly noticeable performance issues, potentially causing the screen to completely freeze for a period of time or show other visual artifacts (e.g., jitter). If a lost/delayed packet causes the loss of a key frame (i.e., I-frame), then the decompressor on the client device will lack a reference for all of the P-frames that follow until a new I-frame is received. Similarly, if a P frame is lost, that will impact the P-frames that follow. Depending on how long it will be before an I-frame appears, this can have a significant visual impact. (As is well-known, I-frames are the only type of frame that is not coded with reference to any other frame. P-frames are coded predicatively from a previous I-frame or P-frame; B-frames are coded predicatively from I-frames and P-frames. In order to be properly decoded, a B-frame associated with a group of pictures (“GOPs”) may need to reference the I-frame of a next GOP. In the context of the present disclosure, the term “I-frame” is intended to broadly refer to an Inter-frame and its equivalents, e.g., an IDR frame in the case of H.264.)
Another problem that arises when transmitting a packet stream at full data rate is that sometimes video frames are dropped or skipped (both terms are used synonymously in the present disclosure) because the bandwidth budget is exceeded by previous frames. Dropped or skipped frames are never encoded. Even though the encoded stream remains contiguous, dropping or skipping frames can still result in visual artifacts that adversely affect the end-user's viewing experience.